Lahar: Warehousing Markovian Streams

Lahar: Warehousing Markovian Streams Julia Maureen Letchner Chair of the Supervisory Committee: Professor Magdalena Balazinska Computer Science and Engineering A huge amount of the world’s data is both sequential and low-level. Many applications consume higher-level information, such as words and sentences, that is inferred from low-level sequences such as raw audio signals using a model (e.g., a hidden Markov model). This inference process is typically statistical, resulting in high-level streams that are imprecise. These imprecise streams, once archived, are useful for analytics support including sequence-finding event queries (e.g. “Find all times when the phrase ‘Barack Obama...veto’ occurs in the NPR news podcast from July 9.”), event query aggregates (e.g. “How many times do 2008 NPR podcasts use the phrase ‘Barack Obama...veto’?”), and event query lineage (e.g. “What words appeared between the word ‘Obama’ and ‘veto’ in the previous query?”). These queries are difficult to support efficiently because archives can be large, and standard relational warehouses cannot support analytics on the rich semantics of imprecise sequences; however, these analytics are critical for allowing applications to effectively leverage this data. In this thesis, we introduce Lahar, the first database system for a common type of imprecise, sequential model called a Markovian stream. Lahar includes novel algorithms for efficiently processing aggregated event queries, and event query lineage. Lahar accelerates performance and scalability of all queries using several techniques, including a set of novel Markovian stream indices and novel methods for approximating Markovian streams. Through experiments on two real-world datasets (one collected from an office-building RFID deployment and the other collected from audio podcasts) we demonstrate that Lahar is an efficient Markovian stream warehousing system.